Repository navigation
ci(bot): enable distillation and cross-review - #620
Conversation
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Nit
Looks good: 2 low provenance issues in the seeded lessons. Splitting the reviewer into a trusted-base checkout and a separate read-only head checkout, plus the same-repo open-PR gate, looks right. Follow-up, not from this PR: engineer-bot-learning.yml still reads inputs.window-hours without bracket notation.
This overview is Claude Opus 5.5's initial take; some of its points were set aside by the cross-check judge — see the findings below.
Other findings
- ⚪ Nit — L-0051 and L-0053 cite #500, which is the legacy learning PR and not where the lesson came from. The source was author runs (#42301071194 and #42966486376). Cite the run IDs or use whatever author-run provenance format the engine accepts. This only affects provenance, not behavior. (Found by Claude Opus 5.5 · confirmed by the judge)
🔀 1 finding considered and dismissed by the judge
- 🔵 Low
.claude/knowledge/lessons.md:30— Some seeded lesson counts don't match the PRs they cite. L-0024 says2×but lists three PRs (#514 #506 #528), L-0033 says2×with three, and L-0066 says1×<br>Judge's evidence:- L-0066 (1×, #566 #562): Serialize branch-editing bot flows with one per-PR concurrency group and restore dependency-se…`
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 Nit
Looks good, with one low provenance note. The reviewer workflow's split into a trusted checkout and a separate review checkout is sound: it checks that the PR is open and not from a fork before checking out the head. The distill config and the frozen archive are consistent.
Other findings
- ⚪ Nit — L-0037 is counted 2× but cites only #565. Its other source is the 2026-11-05 author-run entry, which has no PR number. L-0047 (2×, #529) has the same gap for its 2026-11-07 hang-test entry, while L-0051 already uses a
log:source. Addlog:2026-11-05to L-0037 andlog:2026-11-07to L-0047 so both counts can be traced. (Found by Claude Opus 5.5 · confirmed by the judge)
|
Keeping the supported source format: the engine grammar permits up to three PR references or one log date; mixing |
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall, with 1 low concern: one of the 66 seeded lessons states a GitHub Actions rule that the repo's own workflow contradicts. Splitting the reviewer into a trusted base-SHA checkout plus a read-only head checkout, and the new open/non-fork PR check for manual dispatch, look sound. Follow-up, not caused by this PR: the reviewed-PR checkout is shallow, so PR_BASE_SHA is not in that tree in case the engine ever needs local history.
| - L-0059 (1×, #611): Every auxiliary HTTP request, including OAuth, exchange and CloudFetch, must use the connection provider's getAgent to honor proxy/TLS settings. | ||
|
|
||
| ## Bot workflow and CI | ||
| - L-0060 (2×, #499): Use explicit bracket notation for hyphenated dispatch input names and string-typed numeric knobs in workflows also triggered by schedule. |
There was a problem hiding this comment.
🔵 Low — L-0060 turns the #499 archive entry into a standing rule that both bots will now load: hyphenated dispatch inputs must use bracket notation. That rule is shaky. GitHub Actions expression syntax lets property names contain - under dot dereference. This repo's own learning workflow, which this PR touches, still reads the input as ${{ inputs.window-hours }} with no brackets. So the seeded lesson disagrees with the code it sits next to. Bots will likely raise findings or make edits based on it, and since lessons.md is now the only learning text injected into prompts, a wrong lesson costs more than a wrong archive note. Before the first distill run, either confirm the claim and switch the workflow to inputs['window-hours'], or reduce L-0060 to the part that is verified: use string-typed numeric inputs when the workflow also has a schedule trigger.
There was a problem hiding this comment.
Correction proposed in #621: remove the unsupported bracket-notation requirement from L-0060, retain the string-typed numeric-input guidance, and reduce the occurrence count from 2 to 1 for the retained observation. The engine parser/renderer and actionlint pass.
Apply kernel #476's onboarding: enable distill mode with learning-PR auto-merge off, and pin the bot engine to 6014478 for two independent reviewers plus a judge. Manual reviews use trusted workflow code and a separate resolved PR checkout.
Preserve all 89 rules from #500 at 8e6b681 as a frozen archive and seed 66 lessons. New learning uses dated journal files and a fresh rolling branch.
98 focused engine tests passed. Config, archive preservation, lesson routing/source checks, seven reviewer-input cases, Bash syntax and actionlint passed. Live cross-review approved the current head; source-format nit explained.
After merge, run learning once and inspect its first journal/lessons PR. Check #500 for newer entries before retiring that legacy PR.
Lint passes. Security Scan flags
picomatch@2.3.2(GHSA-fc5q-39xc-vjpf); package manifests and lockfile are unchanged.